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The decays B fi'^H , which proceed via the 

flavor-changing neutral-current (FCNC) process b — )• 
s/i/z, are considered among the most promising probes 
of the standard model (SM) and its extensions. In 
the SM they occur through higher order amplitudes, 
though additional processes with beyond-the-standard- 
model (BSM) contributions could arise. One can obtain 
sensitivity to BSM physics from precise measurement of 
the decay ampUtudes, especially the angular distribu- 
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tions of the decay products. 

The full differential decay distribution for the decay 
B K*{892)fi+fi^ KiTfi+fi^ is described by four 
independent kinematic variables: the dimuon invariant 
mass squared q"^ = Af^^c^(GeV^/c^), the angle 0^ be- 
tween the /i+ (m ) direction and the direction opposite 
to the B (B) meson in the dimuon rest frame, the angle 
Ok between the kaon direction and the direction opposite 
to the B meson in the K* rest frame, and the angle (j) be- 
tween the two planes formed by the dimuon and the K-n 
systems. The angle (j) is zero if the two planes are par- 
allel. The distributions of 6^, 9k, and are projected 
from the full differential decay distribution and can be 
parametrized with four angular observables, Apg, F^, 
A^^\ and U^ : 

f - I + J (1 - Fl){1 - cos^ Ok), 

1 a cos Ok 2 4 



predicts the quantity Aim to be very close to zero for all 
accessible g^, and this quantity is particularly sensitive 
to the existence of CP violation in right-handed currents 
in BSM physics [i,S]. 

BABAR [13], Belle [13], and CDF {l^ have reported 
measurements of ApB and Fl in the B — ?> K*M decay 
modes. All experiments find Aps to be larger than the 
SM expectation, but so far none has sufficient sensitivity 
to be conclusive. In this Letter, we report measurements 
of the angular distributions in the decay B — > K'--*^ pL~ 
using a data sample, corresponding to an integrated 
6.8 fb~^, of pp collisions at a center-of-mass energy of 
^/s = l.m TeV, collected with the CDF II detector be- 
tween March 2002 and June 2010. The measurement up- 
dates and supersedes an earlier analysis on 4.4 fb~^ [14| . 
Besides the 54% increase in luminosity, we have added 
new decays channels B^ — >■ K*~^ fi~ and improved the 



1 dV 



r dCOS On 



^ Pl(1 - cos2 e^) + 1(1- Fl){1 + cos2 0^) 



1 dV 

fd6 



1 

2^ 



+Afb cos 6*^, 
1 + ^(1 - cos 20 + A^m sm2(j) 



,(1) 



where F = r{B — ^ K*fi+fj, ), Aps is the muon 
forward-backward asymmetry, F^ is the K* longitudinal 

(2) 

polarization fraction, Arp is the transverse polarization 
asymmetry |1;] , and Aim is the time- reversal-odd charge- 
and-parity asymmetry (T-odd CP asymmetry) [2|, @1 ■ In 
the case of B decay, sin (f> has opposite sign to that of the 
B decay due to the flipped dimuon plane. Since Aim is a 
coefficient of the sin 2(/) term in Eq. ([l}. Aim is a measure 
of the CP asymmetry. Experimental access to these an- 
gular observables extends the sensitivity to a large class 
of BSM physics models 

The differential decay distribution is calculated in an 
operator product expansion 7]. In the SM, the nonzero 
terms are parametrized by the short-distance Wilson co- 
efficients C^^g iQ. Since various BSM models predict cou- 
plings to the if*" helicity states that are different from 
the SM ones, the measurement of F^ could constrain 
BSM parameters [ij. The SM prediction in the range 

" The 



1 < g2 < 6GeV7c2 is 



pSM _ 
— 



0.73 



-1-0.02 
-0.03 



quantity ApB is expected to be small at low q'^ and 
to have a large positive value at high q^ in the SM. 
In the range 1 < < 6GeV^/c^, the SM predicts 
Af§ = 0.022 ± 0.028 3]. The BSM contributions can 
change the magnitude and the sign of Aps- For exam- 
ple, some BSM models such as supergravity models with 
large tan /3 ^8] allow an opposite sign of C^^ compared 
to the SM, resulting in the opposite sign of ApB at low 
q^. In the SM A^-* is expected to be approximately zero 
at low q^ and negative at high q^ Some BSM mod- 
els like i?-parity conserving minimal supersymmetry [lol | 
predict the existence of right-handed currents. In such a 
case, A^^' could be enhanced up to ± 1 [l|, d, [llj . The SM 



*o 



/i^^ sig- 



selection achieving a 9% increase in B^ — )■ K 
nal efficiency with almost the same background rejection. 
The resulting factor of 82% increase in signal statistics 
allows us to access for the first time the angular observ- 
ables A^"^ and Aim in the decays B — ^ K*fi^ii^. The 
measurements of the total and differential branching ra- 
tios of the B — )• K^*^n'^ii~ decays are reported in another 
letter 

The reconstruction of the B — >■ K^*^ jX^ ^~ decays 
starts with a dimuon sample selected by the online trig- 



ger system [16| of the CDF II detector [17[ . The trigger 
system uses information from muon detectors and the 
central drift tracking chamber. The central drift track- 
ing chamber [l8| provides 96 measurements per track 
between radii of 40 and 137 cm, allowing an accurate 
determination of the charged-particle momentum. The 
CMU and CMX muon drift chambers [l3| cover the pseu- 
dorapidity regions |ry| < 0.6 and 0.6 < \'q\ < 1.0, re- 
spectively [20|. The CMP muon system is located radi- 
ally behind the CMU and an additional steel absorber, 
and covers \r]\ < 0.6. The dimuon trigger requires a 
pair of oppositely charged tracks with momenta trans- 
verse to the beamline pr > 1.5GeV/c, that are also 
identified in the CMU or CMX chambers. At least 
one of the muon tracks in the pair is required to be a 
CMU muon. The trigger also requires that either the 
dimuon pair satisfies L^-j, > 100 /xm, where the trans- 
verse decay length L^y is the flight distance between the 
dimuon vertex and the event primary vertex and pro- 
jected on the dimuon momentum vector, or that one of 
the muon candidates has pr > 3.0GeV/c and is identi- 
fied by both CMU and CMP chambers. The other de- 
tector subsystems relevant for this analysis are discussed 
in Ref. [2l[. Each offline track is required to satisfy the 
standard quality requirements (e.g. hits in the silicon 
detector) to ensure well measured momenta and decay 
vertices [l3l- The decay length and invariant mass of 
each dimuon pair are calculated after a vertex fit using 
the muon tracks. Dimuons are required to have q^ val- 
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ues outside the ranges of 8.68 < q^ < 10.09 GeVVc^ and 
12.86 <q^ < 14.18 GeVVc^ PJ, to be inconsistent with 
decays of either J/il) or ')p(2S) mesons, which are typi- 
cahy reconstructed with 14 MeV/c^ mass resolution. The 
dimuon pair is then combined with charged tracks form- 
ing a K*° — > K+TT^ candidate to form a -B*^ — > A'*"/i+^^ 
candidate, or a — > Kg{-^ 7r"'"7r^)7r+ candidate to 
form a 5+ K*~^^^fj,^ candidate. Charge-conjugate 
modes are imphed throughout this Letter. The Kg, K*'^, 
and K*~^ candidates are required to have reconstructed 
invariant masses consistent with the world average val- 
ues [i^], and to have > 1 GeV/c. The Kg is also re- 
quired to have its decay vertex displaced from the dimuon 
vertex. The ambiguity in the mass assignment of the de- 
cay products in the K*'^ K^ir^ decay is handled by 
choosing the combination whose K^n^ mass is closest to 
the world average K*'^ mass, which is correct in approxi- 
mately 92% of cases. The reconstructed B candidates are 
required to have pt > 4GeV/c. To further optimize the 
event selection, an artificial neural network (NN) clas- 
sifier is trained for each channel using simulated signal 
events and background events taken from mass sidebands 
in data. Simulated signal distributions are corrected us- 
ing scale factors extracted by comparing simulation to 
data in the control samples B — J/'0(— /i+/i~)_fi'(*'. 
The optimized NN threshold is determined to minimize 
the statistical uncertainty of the angular observables, 
using many kinematic observables including px, invari- 
ant mass, vertex fit parameters, and muon identification 
quahty [ll |. 

The signal yield is obtained by an unbinned maximum 
likelihood fit to the invariant mass distribution with a 
probability density function (PDF) consisting of Gaus- 
sian distributions for the signal and a linear background. 
We find a total of 234 ± 19 B+ ^ K+^+n-, 164 ± 15 
B° K*°n+n-, and 20 ± 6 B+ ^ K*+ n+ events. 
We divide the signal region into six bins in . Two semi- 
inclusive bins are included with ranges covering theoreti- 
cally well-controlled regions. We obtain the signal yields 
in the individual q^ ranges by fitting the invariant mass 
in a similar way. The resulting yields are listed in Ta- 
bles U HH and mil The invariant mass distributions are 
shown in Fig. [TJ 

To extract the quantities F^, ApB, Aj^ , and Aim, we 
perform likelihood fits to distributions of cos6'i<-, cos 6*^, 
and ip for events in each q'^ range. The signal fractions 
are fixed to the values obtained from the invariant mass 
fits. Signal PDFs for angular distributions are formed 
from Eq. ([1]), taking into account the estimation of the 
detector acceptance of the decay angles and the K-n in- 
terchange using Monte Carlo simulations. The incorrect 
K-TT assignment in the K*'^ — >■ K~^Tr~ decay distorts the 
signal mass distribution and swaps the sign of cos^^. 
This effect is taken into account by introducing an addi- 
tional signal-like term into the likelihood function. The 
contribution from decays with non-resonant K-tt pairs is 




MiiniK") (GeV/c^) M(finK"*) (GeV/c^) 



FIG. 1: Invariant mass of B° K*°^j.+fi- (left) and B+ -5> 
K*'^ fi^ (right), with fit results overlaid. 

estimated to be small [l[ and neglected in the fit. The 
background PDF shapes for the angular distributions are 
modeled from events in the B mass sidebands. The val- 
ues of F]^ in individual ranges are extracted from fits 
to the COS0K distributions and then used as inputs in 
the determinations of the other three observables. The 
asymmetry Aps is obtained from fits to the cos0^ dis- 

(2) 

tributions, and A}p and Aim arc from fits to the (j> dis- 
tributions. 

The extracted observables in the decay mode B'^ ^• 
K*^lX^ li~ are listed in TableHl and the forward-backward 
asymmetry ApB is illustrated in Fig. [Sfa) as a function 
of q^ . To increase sensitivity, we also perform a fit to the 
combined i?° -J> iir*°/i+^~ and i?+ i4r*+/i+^" modes, 
assuming they have the same decay dynamics. The com- 
bined fit results are listed in Table HIl and shown in Fig.|31 
On average ApB resolutions are improved by factors of 
1.5 (1.8) times from previous CDF measurement |1J] for 
5° K*°^+^i- {B K*n+^- ). The current data are 
consistent with the SM and with the inverted C7 sce- 
nario. In addition, a new measurement of ApB in the 
decay — > K^/i^fi^ is obtained. In the SM, the ex- 
pected value of ^ PR for this mode is quite small over the 
entire range of [23|, whereas some BSM models predict 
enhanced values of ^HJ- In the ApB fit, we assume 
no scalar term [23] and set Fl = 1. The result is shown 
in Fig. m^b) and listed in Table IIIII The current data is 
consistent with the SM. 

The sources of systematic uncertainty in the angular 
observables include the estimation of detector acceptance 
of the decay angles, signal fraction estimation and shape 
modeling of events in the signal window, feed-down back- 
ground from other B decays, trigger efficiency and bias 
modeling, incorrect K-tt assignment in the if*" — ?> K'^tt^ 
decay, and fitting bias. The largest contribution is that 
from uncertainties on the signal fraction in the signal win- 
dow. The total systematic uncertainties on the asymme- 
try observables for various q^ values are in the ranges 0.02 
- 0.09 for Fl, 0.05 - 0.17 for ApB, 0.07 - 2.21 for A^^\ 
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FIG. 2: Measurements of forward-backward asymmetry Afb 
in the decay (a) B° — >■ K*'^^'^^" and (b) 5+ — >■ ji^ ^~ as a 
function of dimuon mass squared . Points are the fit results 
from data. The solid curves are the SM expectation [SJ]. The 
dotted curve is the C7 = — Cf^ expectation suggested by 
some BSM models. Hatched regions are excluded resonant 
(charmonium) decay regions. 



and 0.01 - 0.14 for A,™ in the mode B° K*°fi+fi-. 
Similar sizes of systematic uncertainties are found in the 
combined fit results. 

In summary, we have reconstructed the decays — > 
K*'^fi^lJ.^ and 5+ — > K*^fi^iJ.^ and measured their 
angular distributions. We have measured the muon 
forward-backward asymmetry Aps, the K* longitudi- 
nal polarization fraction Fl, the transverse polarization 
asymmetry Aj, , and the T-odd CP asymmetry Aim as a 
function of the dimuon mass squared q^. Measurements 
of Aj, and Aim are reported for the first time. The 
muon forward-backward asymmetry ApB is also mea- 
sured in the decay mode — >■ fi^ and represents 
an update with higher precision. All of the new reported 
results presented in this Letter are among the most accu- 
rate to date and consistent with the SM predictions, but 
still statistically limited in providing stringent tests on 
various models. The results are also consistent with re- 



cent measurements from _B-factory experiments [12l. |13|. 
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FIG. 3: Measurements of (a) longitudinal K* polarization 
fraction Fl, (b) forward-backward asymmetry Afb, (c) trans- 
verse polarization asymmetry aI^'' , and (d) T-odd CP asym- 
metry Aim in the combined decay mode B — > K* fj,^ , all 
presented as a function of dimuon mass squared g^. The 
points are the fit results from data. The solid curves are the 
SM expectation The dotted curves are the Cy = -Cf^ 
expectation. Hatched regions are resonant (charmonium) de- 
cay regions. 
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